[AMD][AgentX] DSv4 MI355X agentic disagg: add EAGLE/MTP speculative-decoding arm / DSv4 MI355X 智能体分离式推理：新增 EAGLE/MTP 投机解码分支 - #2309
Conversation
…fill cudagraph for DSv4 agentic disagg Co-authored-by: Cursor <cursoragent@cursor.com>
…x router policy env var/name Co-authored-by: Cursor <cursoragent@cursor.com>
…ic-hicache Resolves review comment: the image bump (v0.5.14 -> v0.5.15.post1), re-enabled TP8/EP8/DPA hicache arm, and HiCache sizing/NCCL/router fixes need a changelog entry to trigger the benchmark sweep on merge. 中文：为 dsv4-fp4-mi355x-sglang-disagg-agentic-hicache 添加 perf-changelog.yaml 条目。解决评审意见：镶像升级（v0.5.14 -> v0.5.15.post1）、重新启用的 TP8/EP8/DPA hicache 分支，以及 HiCache 容量/NCCL/路由策略修复，需要一条 changelog 条目才能在合并时触发基准测试扫描。 Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
…entX DeepSeek-V4's MTP head is a native NextN/EAGLE draft (shipped with the model), so SGLang defaults the draft path to the main checkpoint and does not need --speculative-draft-model-path. Adds mtp_flags matching the upstream SGLang DeepSeek-V4 AMD recipe (EAGLE, topk=1); num-steps and num-draft-tokens are already derived from DECODE_MTP_SIZE in server_sglang.sh. Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
…ic-hicache-mtp Required per AGENTS.md's "Adding a benchmark configuration" rule so the new EAGLE/MTP arm actually triggers a benchmark sweep on merge. 中文：为 dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp 添加 perf-changelog.yaml 条目。根据 AGENTS.md 的"添加基准测试配置"规则，需要此条目 才能在合并时为新的 EAGLE/MTP 分支触发基准测试扫描。 Co-authored-by: Cursor <cursoragent@cursor.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ichbinblau
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30064907923 |
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30076065135 |
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30079546440 |
SGLang's PD-disaggregated KV/state transfer requires both ends of the connection to register the same number of state descriptors. Gating --speculative-* flags to the decode server only left prefill without the extra MTP/EAGLE draft state component, causing decode's PD state transfer to fail with "state component count mismatch (local=3, remote=4)" as soon as a transfer was attempted on the new disagg-hicache-mtp arm. Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30080234740 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30085878578 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30085878578 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30085878578 |
There was a problem hiding this comment.
thanks for this PR @ichbinblau ! this is very exciting
1 small nit: per https://github.com/SemiAnalysisAI/InferenceX/blob/main/docs/PR_REVIEW_CHECKLIST.md in order to do fair MTP
SGLANG_SIMULATE_ACC_LEN: <AL from yaml>
SGLANG_SIMULATE_ACC_METHOD: match-expected
SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token
https://github.com/SemiAnalysisAI/InferenceX/tree/main/golden_al_distribution
…bs (#2330) The golden AL README only showed vLLM's synthetic_acceptance_length. Add the SGLang env vars (SGLANG_SIMULATE_ACC_LEN / _METHOD / _TOKEN_MODE) and TRT-LLM's TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS, including the off-by-one: the TRT-LLM value counts accepted draft tokens only and excludes the bonus token, so it must be set to golden AL minus 1 (fractional allowed, e.g. AL 3.5 -> 2.5). Also teach the codeowner sign-off verify prompt (Check 10) to accept TRT-LLM configs and to compare the pinned TRT-LLM value plus 1 against the golden curve. Keep README_zh.md in parity. Requested in #2309 (review) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Addresses PR #2309 review feedback (functionstackx): #2309 (review) Agentic trace replay doesn't reproduce real token-by-token traffic, so measured MTP/EAGLE acceptance there isn't representative of production. Per the AgentX fairness guidelines, wire SGLANG_SIMULATE_ACC_LEN (pinned to the committed golden AL in golden_al_distribution/dsv4_mtp.yaml, thinking_on, num_speculative_tokens=3 -> 2.49), SGLANG_SIMULATE_ACC_METHOD and SGLANG_SIMULATE_ACC_TOKEN_MODE into the decode launch command for agentic MTP runs, matching the existing GB300 dynamo-sglang DSv4 MTP recipes. Falls back to real acceptance with a warning if a future draft length has no committed golden AL wired in yet. Co-authored-by: Cursor <cursoragent@cursor.com>
Extend the simulated-acceptance lookup to cover DECODE_MTP_SIZE=1 and =2 using the committed thinking_on curve in golden_al_distribution/dsv4_mtp.yaml (1 -> 1.79, 2 -> 2.27), matching the existing size=3 -> 2.49 entry. Sizes without a wired golden AL still fall back to real acceptance with a warning. Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30227937400 |
Thanks for flagging this @functionstackx — addressed in 5f2bfef and 1f4de33: |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30228333690 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30228333690 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30228333690 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30228333690 |
2 participants
Summary
DSv4 FP4 MI355X SGLang PD-disaggregated agentic-coding benchmark: HiCache sizing/NCCL/router fixes, the TP8/EP8/DPA hicache search-space arm, and a new EAGLE/MTP speculative-decoding arm. Originally split as #2308 (fixes) stacked under #2309 (MTP), now combined into a single PR targeting
maindirectly.What's changed
HiCache sizing (ratio vs size)
dsv4_fp4_mi355x_sglang-disagg.sh: setHICACHE_RATIO=4andFORCE_HICACHE_RATIO=1so this recipe explicitly opts into ratio-based sizing (DeepSeek-V4's hybrid HiCache pool rejects--hicache-size).job.slurm: forwardFORCE_HICACHE_RATIOinto the container's HiCache env file (hicache_mc_<job>.env) alongside the otherHICACHE_*vars -- it was previously silently dropped, so the override never reached the server.NCCL / decode cuda-graph crash workaround
env.sh: addTORCH_NCCL_BLOCKING_WAIT/NCCL_BLOCKING_WAIT=1to avoid a ROCm 7.2 HIP-runtime watchdog race during decode cuda-graph capture ([AMD] qwen3-235b-mxfp4 nightly aborts at CUDA-graph capture: NCCL watchdog hipEventQuery on capturing stream (hipErrorCapturedEvent) sgl-project/sglang#29235, [Block Issue][hip graph]: HIP error: operation not permitted when stream is capturing ROCm/hip#3876, [ROCm] Avoid watchdog event queries during graph capture pytorch/pytorch#176251);NCCL_DEBUG=INFOleft commented out (log-spam reduction).models.yaml: disable prefill cuda-graph (disable_cuda_graph: true) forDeepSeek-V4-Pro-AgentX, drop--enable-two-batch-overlapfromdp_flags, lower prefillmem_fraction_static0.8->0.72 and no-dpmax_running_requests128->64; remove the redundantdisable_cuda_graph: falsefrom decode (already the code default).lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710->v0.5.15.post1-rocm720-mi35x-20260719.RDMA QoS/DCQCN + GPU-sanity pre-flight gates (
job.slurm)helpers/rdma_check.sh: validates NIC QoS (PFC/DSCP) and DCQCN configuration on every allocated node viasrunbefore any container/GPU time is spent. Override withSKIP_RDMA_CHECK=1.helpers/gpu_sanity.sh: after container pre-clean/teardown, fails fast (naming the offending process) if a bare, non-containerized process is still holding GPU VRAM.Router policy fix + DPA (ep=8/dp-attn) hicache arm
server_sglang.sh(agentic branch only -- DI branch untouched): fixes a bug where the script readROUTER_PREFILL_POLICYbut recipe scripts exportPREFILL_ROUTER_POLICY, so the recipe's policy override was silently ignored and the router always fell back to the hardcoded default. Also switches the default prefill router policycache_aware->consistent_hashing, adds--dp-awareto the router flags, and comments out the separate--decode-policyoverride (superseded by--dp-aware).PREFILL_DP_ATTN/DECODE_DP_ATTNto the env-var whitelist forwarded into the sibling client container, alongside the existingPREFILL_ENABLE_DP/DECODE_ENABLE_DP.configs/amd-master.yaml: bumpdsv4-fp4-mi355x-sglang-disagg-agentic-hicacheimage tov0.5.15.post1-rocm720-mi35x-20260719; re-enable (previously commented out) the TP8/EP8/DPA (ep=8, dp-attn=true) hicache search-space arm atconc-list: [64, 96, 128].trace_replay.sh: whenPREFILL_ENABLE_DP=true, exportAIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=trueso aiperf's HTTP routing key is derived from the correlation ID under DP-attention.EAGLE/MTP speculative-decoding arm
models.yaml: addmtp_flags: "--speculative-algorithm EAGLE --speculative-eagle-topk 1"toDeepSeek-V4-Pro-AgentX. DeepSeek-V4's MTP head is a native NextN/EAGLE draft shipped with the model, so SGLang defaults the draft path to the main checkpoint -- no--speculative-draft-model-pathneeded.--speculative-num-steps/--speculative-num-draft-tokensare already derived fromDECODE_MTP_SIZEinserver_sglang.sh;mtp_flagsis only appended to the decode server config (MTP is decode-only).configs/amd-master.yaml: newdsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtpconfig-key with two search-space arms mirroring the non-MTP hicache config's topologies (spec-decoding: "mtp",DECODE_MTP_SIZE=3): TP8/EP1 no-DP atconc-list: [2, 4, 8, 16, 32], and TP8/EP8/DPA atconc-list: [64, 96, 128].perf-changelog.yamlTwo appended entries (tail of file, per the append-only convention), one per config-key touched, both linked to this PR:
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache(image bump / HiCache / NCCL / router fixes / DPA arm) anddsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp(new EAGLE/MTP arm).Test plan
bash -non all touched shell scripts (env.sh,job.slurm,server_sglang.sh,trace_replay.sh,setup_deps.sh,helpers/gpu_sanity.sh,helpers/rdma_check.sh) -- syntax OKconfigs/amd-master.yaml,benchmarks/multi_node/amd_utils/models.yaml,perf-changelog.yaml-- YAML parses cleanlygenerate_sweep_configs.py full-sweepresolves all 16dsv4-fp4-mi355x-sglang-disagg-agentic-hicache*configs correctly: 8 non-MTP (TP8/EP1 conc 2/4/8/16/32, TP8/EP8/DPA conc 64/96/128) + 8 MTP twins with identical topologies plusspec-decoding: mtpmainto confirm no unintended regression in theIS_AGENTIC/DI router split or the router-readiness canary (wait_or_die)e2e Test - dsv4-fp4-mi355x-sglang-disagg-agentic-hicacheand...-hicache-mtp(multi-node, mi355x-amds)中文说明
DSv4 FP4 MI355X SGLang 预填充/解码分离式（disaggregated）智能体编码（agentic-coding）基准测试：修复 HiCache 容量配置、NCCL 以及路由策略问题，重新启用 TP8/EP8/DPA 的 hicache 搜索空间分支，并新增一个 EAGLE/MTP 投机解码（speculative decoding）分支。原本拆分为 #2308（修复项）叠加在 #2309（MTP）之上，现已合并为一个直接面向
main的单一 PR。HiCache 容量配置（ratio 与 size）
dsv4_fp4_mi355x_sglang-disagg.sh：设置HICACHE_RATIO=4和FORCE_HICACHE_RATIO=1，使该配方显式启用基于比例（ratio-based）的容量配置（DeepSeek-V4 的混合 HiCache 池不支持--hicache-size）。job.slurm：将FORCE_HICACHE_RATIO转发到容器的 HiCache 环境变量文件（hicache_mc_<job>.env）中，此前该变量被静默丢弃，导致覆盖配置无法生效。NCCL / 解码 CUDA-graph 崩溃的规避方案
env.sh：新增TORCH_NCCL_BLOCKING_WAIT/NCCL_BLOCKING_WAIT=1，以规避 ROCm 7.2 在解码 CUDA-graph 捕获阶段出现的 HIP 运行时看门狗（watchdog）竞态问题；NCCL_DEBUG=INFO保持注释状态以减少日志噪音。models.yaml：为DeepSeek-V4-Pro-AgentX禁用预填充 CUDA-graph（disable_cuda_graph: true），从dp_flags中移除--enable-two-batch-overlap，将预填充mem_fraction_static从 0.8 降至 0.72，非 DP 场景下max_running_requests从 128 降至 64；移除解码侧多余的disable_cuda_graph: false（已是代码默认值）。lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710->v0.5.15.post1-rocm720-mi35x-20260719。RDMA QoS/DCQCN 与 GPU 健康检查前置校验（
job.slurm）helpers/rdma_check.sh：在占用任何容器/GPU 时间之前，通过srun校验每个分配节点的网卡 QoS（PFC/DSCP）与 DCQCN 配置；可通过SKIP_RDMA_CHECK=1跳过。helpers/gpu_sanity.sh：容器预清理/收尾之后，若仍有裸机（非容器化）进程占用 GPU 显存，则快速失败并报告具体进程。路由策略修复 + DPA（ep=8/dp-attn）hicache 分支
server_sglang.sh（仅智能体分支，DI 分支未改动）：修复脚本读取ROUTER_PREFILL_POLICY而配方实际导出的是PREFILL_ROUTER_POLICY的问题，此前该覆盖配置被静默忽略，路由器始终回退到硬编码默认值。同时将默认预填充路由策略由cache_aware切换为consistent_hashing，为路由参数新增--dp-aware，并注释掉原有的--decode-policy覆盖项（已被--dp-aware取代）。PREFILL_DP_ATTN/DECODE_DP_ATTN加入转发到客户端容器的环境变量白名单，与现有的PREFILL_ENABLE_DP/DECODE_ENABLE_DP并列。configs/amd-master.yaml：将dsv4-fp4-mi355x-sglang-disagg-agentic-hicache镜像升级至v0.5.15.post1-rocm720-mi35x-20260719；重新启用此前被注释的 TP8/EP8/DPA（ep=8, dp-attn=true）hicache 搜索空间分支，并发列表为[64, 96, 128]。trace_replay.sh：当PREFILL_ENABLE_DP=true时，导出AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true，使 aiperf 的 HTTP 路由键在 DP-attention 下基于关联 ID（correlation ID）派生。EAGLE/MTP 投机解码分支
models.yaml：为DeepSeek-V4-Pro-AgentX新增mtp_flags: "--speculative-algorithm EAGLE --speculative-eagle-topk 1"。DeepSeek-V4 的 MTP 头是模型自带的原生 NextN/EAGLE 草稿模型，因此 SGLang 默认将草稿路径指向主检查点，无需--speculative-draft-model-path。--speculative-num-steps/--speculative-num-draft-tokens已在server_sglang.sh中由DECODE_MTP_SIZE派生；mtp_flags仅追加到解码侧服务器配置中（MTP 仅作用于解码）。configs/amd-master.yaml：新增dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp配置项，包含两个搜索空间分支，拓扑结构与非 MTP 版本的 hicache 配置一致（spec-decoding: "mtp"，DECODE_MTP_SIZE=3）：TP8/EP1 非 DP，并发列表为[2, 4, 8, 16, 32]；以及 TP8/EP8/DPA，并发列表为[64, 96, 128]。perf-changelog.yaml在文件末尾追加两条记录（遵循只追加约定），分别对应本次改动涉及的两个配置项，均关联本 PR：
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache（镜像升级/HiCache/NCCL/路由修复/DPA 分支）与dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp（新增 EAGLE/MTP 分支）。测试计划
env.sh、job.slurm、server_sglang.sh、trace_replay.sh、setup_deps.sh、helpers/gpu_sanity.sh、helpers/rdma_check.sh）执行bash -n—— 语法正确configs/amd-master.yaml、benchmarks/multi_node/amd_utils/models.yaml、perf-changelog.yaml—— YAML 解析无误generate_sweep_configs.py full-sweep正确解析全部 16 个dsv4-fp4-mi355x-sglang-disagg-agentic-hicache*配置：8 个非 MTP（TP8/EP1 并发 2/4/8/16/32，TP8/EP8/DPA 并发 64/96/128）+ 8 个拓扑相同并带spec-decoding: mtp的 MTP 对应版本main分支对比，确认IS_AGENTIC/DI 路由拆分逻辑及路由就绪校验（wait_or_die）未出现意外回归e2e Test - dsv4-fp4-mi355x-sglang-disagg-agentic-hicache与...-hicache-mtp（多节点，mi355x-amds）Made with Cursor